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ABSTRACT 



This paper presents the results of the work of nine teacher 
researchers who took an in-depth look at instructional practices and beliefs 
about oral language and its assessment. There are three sets of research 
questions: (1) How do teachers and students believe oral instruction should 

be handled in the classroom and to what extent do the beliefs that teachers 
have about students' oral production in the classroom and the students' own 
beliefs match the reality? (2) How are teachers using the required materials 
for oral language production? and (3) How do teachers assess students orally? 
Forty teachers and 63 students at all levels of a Colombian university's 
adult English program participated in this study. It was found that both 
teachers and students generally believed successful oral production was based 
on accuracy. Teachers' believed instruction was more student-centered, which 
was inconsistent with their actual classroom practice. Consistent with 
student beliefs, student evaluation was in fact accuracy-based. The 
pedagogical implications included a need for: (1) guidance both to teachers 

and students in how to make EFL teaching and learning more truly 
communicative in nature; and (2) more teacher training in how to conduct 
language assessment in the classroom. Appendices include: "Beliefs/Materials 
Questionnaire"; "Assessment Questionnaire"; "Beliefs/Materials Follow-Up 
Questionnaire"; "Assessment Follow-Up Questionnaire"; "Materials Group 
Interview"; and "Observation Tool." (Contains 17 references.) (KFT) 
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ABSTRACT 



Given the myriad of reasons why EFL students may not develop speaking skills, 
there has emerged a clear need for research on beliefs and practices regarding the 
teaching, learning, and assessment of speaking. In the spirit of action research, nine 
Colombian teacher-researchers took a bottom-up, in depth look at instructional 
practices and beliefs about oral language and its assessment. Their research questions 
were: 

1 . How do teachers and students believe oral instruction should be handled in 
the classroom? To what extent do the beliefs that teachers have about 
students’ oral production in the classroom and the students’ own beliefs 
match the reality? 

2. How are teachers using the required materials for oral language production? 

3. How do teachers assess students orally? 

Forty teachers and 63 students in beginning, intermediate, and advanced 
courses at a private Colombian university’s adult English program participated in the 
study. Instrumentation included questionnaires, follow-up questionnaires, interviews, 
and classroom observation. 

The teacher-researchers found both teachers and students generally believed 
successful oral production was based on accuracy. While both teacher and student 
beliefs about the appropriate ratio of teacher talk to student talk reflected a teacher- 
centered classroom, teachers’ beliefs somewhat more of a student-centered approach - 
inconsistent with their actual classroom practice. In addition, while the frequency and 
manner of teacher feedback varied, characteristics of oral language considered when 
assessing students orally were accuracy-based, rather than reflecting a communicative 
approach. 

The pedagogical implications included a need for (1) guidance both to teachers 
and students in how to make EFL teaching and learning truly communicative in nature, 
and (2) more teacher training in howto conduct language assessment in the classroom. 



Introduction 



The teaching and assessment of oral language in the university-level English as 
a foreign language classroom in some parts of the world continues to be a challenging 
endeavor. Teachers are frequently not fluent in English and therefore not confident in 
their use of the language. The EFL students do not necessarily speak very much 
English in class, nor do they have much opportunity to practice using the language 
outside of the classroom. Classes may be large, the curriculum may favor the written 
language, and the focus may be on grammar rather than on oral communication. 

Hence, the speaking skills among these EFL students may not be well exercised and 
consequently underdeveloped. 

Commensurate with a lack of emphasis on speaking instruction, there may be a 
reluctance on the part of teachers to assess oral language in the classroom. Aside from 
the issues of time and logistics, a plausible explanation is that the EFL teacher does not 
receive adequate training or exposure to how to assess oral language performance so 
as to feel comfortable doing it. Along with a possible lack on the part of teachers to 
promote speaking in the classroom and a sense of inability on the part of EFL students 
to speak English adequately, there may also be a set of beliefs on the part of both 
students and teachers supporting a more traditional approach to language instruction - 
that a reasonable way to proceed is to focus on the other modalities (that is, grammar, 
reading, and writing) rather than on oral communication. A logical explanation for this 
would be the special demands that oral communication puts both on the nonnative 
English-speaking teachers who must serve as a model of English fluency and on the 
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students as well, who are called upon to perform orally in front of their peers and 
possibly lose face as a result. 

A Brief Review of Literature 

The current popularity of communicative approaches to ESL and EFL instruction 
in many parts of the world has prompted teachers to look for varied means of assessing 
their students’ oral abilities in the classroom (see Brown, 2001, Ch. 3). 2 With this shift in 
the focus of oral language assessment from more traditional interviews with pat 
questions to more communicative, performance-oriented measures (see, for example, 
Brown, 1998, and Norris, Brown, Hudson, & Yoshioka, 1998), perhaps there will be an 
increased trend in foreign-language teacher training programs around the world towards 
these more performance-oriented measures. As these measures are more flexible and 
offer more possibilities for how to evaluate oral performance, teachers may feel they are 
more accessible and may be more comfortable with them. 

Much of the research on teacher and student beliefs in foreign language 
classrooms has so far relied primarily on Horwitz’s “Beliefs about Language Learning 
Inventory” (BALLI) (1983, 1987), which includes 34 broadly-tuned items, including items 
relating to various aspects of speaking, such as beliefs about the ease of learning to 
speak, the importance of pronunciation, committing errors in speaking, and the role of 
practice. Studies using the BALLI, have, for instance, compared the beliefs of Russian 
learners of English to those of American learners of French and Spanish (Tumposky, 

2 Brown offers six interconnected characteristics as a description of communicative language teaching - 
focusing on all components, engaging learners in use of language for meaningful purposes, striking a 
proper balance between fluency and accuracy, teaching for out-of-class communication, focus on the 



1991). Interestingly, in that study, the 54 Russian students were more likely to hold the 
belief that it was important to take risks and to practice speaking the language, while the 
36 Americans were holding themselves back from practicing the speaking skill, although 
they were motivated to achieve fluency. These American students believed that the 
learning of the target language was not viewed by their compatriots as an important or 
valued achievement, nor would it necessarily lead to better employment opportunities. 

It should also be pointed out that the Russian students were a select group of 
undergraduates who were in the U.S. on an orientation program before being placed as 
exchange students in American colleges, so they had already committed themselves to 
risk taking and were in an ESL, not an EFL, situation. 

A more recent study by Kern (1995) demonstrated how the BALLI can be used to 
compare teacher and student beliefs and to tease out differences that may exist. In a 
study of the beliefs of 288 first and second semester college French students and their 
instructors, it was found that in certain domains teachers' beliefs bore little, if any, 
relationship to students’ beliefs. For example, students’ and instructors' opinions on 
pronunciation, error correction, and the importance of rule learning contrasted more at 
the end of the semester than at the beginning. Kern highlights the importance not only 
of the nature of the textbook but also of the test materials. As he puts it, “In the final 
analysis it is not what we say that is important or unimportant, but rather what we 
assess, and how we assess it, that will send a clear message to our students about 
what instructed language learning is all about” (Kern, 1995: 81). 



learning process, teacher as facilitator (Brown, 2001, p. 43)). 
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The Need for this Study 

Given the myriad of reasons why EFL students may not develop speaking skills 
and why the assessment of these abilities may be limited, there has emerged a clear 
need for research on beliefs and practices regarding the teaching, learning, and 
assessment of speaking. In addition, since the issues are so deeply embedded in 
classroom practices, it would seem imperative to engage classroom teachers in the 
research effort. Hence, a plan for study would entail both quantitative and qualitative 
forms of action research by teachers, including the design of interview schedules and 
questionnaires, and the collection of data (Hopkins, 1993; Seligerand Shohamy, 1989; 
Freeman, 1998; Wallace, 1998; Burns, 1999). Wallace defines action research as 
"systematically collecting data on your everyday practice and analyzing it in order to 
come to some decisions about what your practice should be” (1998: 4). Wallace also 
underscores the benefits of collaborative action research - that is, working in 
subgroups of teacher colleagues. 

The study reported on in this article reflects a research project involving issues 
that were raised by teachers at the grass-roots level. It reflects action research in the 
true sense of the word where local teachers in an English as a Foreign Language 
program met together repeatedly until a research study emerged. In response to this 
need for research on beliefs and practices regarding the teaching, learning, and 
assessment of speaking, nine Colombian teacher-researchers took an in-depth look at 
instructional practices and beliefs about oral language and its assessment. The study 
arose out of an awareness that in order to be more competitive on a global scale, 
citizens of third world countries need to be proficient in English and as a response to a 



1994 mandate by the Colombian government that students be bilingual in order to 
procure a professional degree. Because language instruction in the past had not 
emphasized listening and speaking, the local teachers got together to do research on 
the oral component of the their EFL language classes. 

The Language Center in a private university in Medellin, Colombia, set up a 
Research and Development Unit in order to engage in research on the practices at their 
center. 3 Initially, a group of fifteen or so teachers enrolled in an in-service training 
course engaged in self-inquiry in their classrooms, and conducted an error correction 
project in which they examined the types of error correction that were taking place in 
their classrooms. Then a number of those teachers were hired to devise research 
projects consistent with the center's goals of internationalizing the curriculum and 
implementing a more communicative approach to language teaching. 

One of the outcomes of their self-examination was the realization that although 
teachers wanted to add an oral component to their classes in an attempt to adopt a 
communicative style of teaching, they lacked the knowledge with which to do it. 

Classes consisted primarily of teacher talk. When students did communicate orally, it 
was generally with the teacher rather than with other students. There was very little 
interaction or group work. This teacher researcher unit ultimately settled on three areas 
of concern related to oral production in the classroom that they wished to investigate: 
students’ and teachers’ beliefs about oral production in the classroom, materials used 

3 The research team was headed by the author of this article, Fass, and included nine 
teacher researchers: Luz Dary Aristizabal G., Marie-Claire Binder de B., Angela 
Campo, Fernando Grespo-Orozco, Sandra O. Gaviria, Luz Adriana Lopera O., Ana P. 
Munoz,’ Marcela Palacio U., and Consuelo Uribe P. The second co-author, Cohen, 
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for providing the oral component of the class, and the measures used in oral 
assessment. 

The research questions were as follows: 

1 . How do teachers and students believe oral instruction should be handled in the 
classroom? To what extent do the beliefs that the teachers have about students’ oral 
production in the classroom and the students’ own beliefs match the reality? 

2. How are teachers using the required materials for oral language production? 

3. How do teachers assess students orally? 

Research Design 

Sample 

The sample was drawn from teachers and students involved with the Adult 
English Program at a private university in an upper-class neighborhood of Medellin, 
Colombia, the second largest city in Colombia. The teacher sample at the Adult English 
Program consisted of fifty-one teachers (plus the nine teacher researchers, who did not 
participate in the study). The vast majority of teachers were native speakers of Spanish 
who grew up in Spanish speaking countries or were bilingual, having lived at least part 
of their childhood in the U.S. These teachers came mostly from upper-class areas and 
the majority had spent time abroad. 

Slightly fewer than half of the EFL teachers were university students who were 
working on their undergraduate degrees. Those with post-secondary degrees came 
from a wide variety of specializations. Only about a quarter of them had undergraduate 



served as an external consultant to the project. 
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degrees in language teaching, others had undergraduate degrees in education, 
although not necessarily in language teaching, and the majority had no training in 
education at all. Those teachers with experience in language education were most 
familiar with the grammar-translation and audio-lingual methodologies since this was 
the way that they themselves had learned languages in school and was consequently 
the method of instruction that most of them employed in their classrooms. 

The student sample consisted of 63 subjects, reflecting 5% of those enrolled in 
beginning, intermediate and advanced level EFL courses. Information was gathered 
from five different beginner classes with a total of 42 student respondents, three 
intermediate classes with 15 students, and two advanced classes with a total of 6 
students. Courses were chosen according to the number of students enrolled in order 
to have the sample mirror as closely as possible the percentage of students at the high, 
intermediate, and low levels in the entire program. Adult courses were attended by both 
employees sent from large companies in the city and students from the surrounding 
wealthier communities. 



Instrumentation 

Beliefs Questionnaires, Interview, and Classroom Observation Instrument 

An initial questionnaire was constructed in Spanish by the three teacher 
researchers in the group focusing on beliefs, referred to from here on as “the Beliefs 
Group.” The group asked those EFL teachers who were participating in the study as 
respondents their beliefs about: (1 ) the ideal percentage of class time for teacher talk 
and why, (2) the ideal percentage of class time for student talk and why, (3) the 
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characteristics of successful oral production by students in a class, and (4) the types of 
oral activities appropriate for learning and practicing English in class. The questionnaire 
was designed so that it could be used for both teachers and students (see 
Beliefs/Materials Questionnaire, Appendix A). 

On the basis of responses from the administration of the first questionnaire 
regarding types of oral activities that were appropriate, the three Beliefs Group teachers 
realized that they needed to know how appropriate teachers felt each type of oral 
activity was. As a result, they generated a list of activities based on those activities 
provided by teachers and students, and created a follow-up questionnaire (see 
Beliefs/Materials Follow-up Questionnaire, Appendix B) in which they asked their 
teacher colleagues to score each activity on a scale of five: from "very appropriate" to 
"not appropriate" for oral practice in the classroom. 

In an effort to determine whether teacher and student beliefs matched the reality, 
a class observation instrument was also designed on the basis of responses from the 
first questionnaire (see Appendix D). The instrument consisted of a chart containing a 
list of types of oral activities based both on responses to the teacher and student 
questionnaires, and on the professional literature: Brown and Yule’s task types (1983), 
Cohen’s suggestions for assessing speaking skills (1994), and Wallace’s (1998) 
observation techniques. The instrument called for identification of all oral activities, and 
an indication of whether they were conducted as a whole class, in pairs, or in groups, 
and for timing of the amount of teacher talk and student talk. 



Materials Questionnaires and Interview 

An initial questionnaire was designed by three teacher researchers (henceforth 
referred to as “the Materials Group”) focusing on the materials used for teaching oral 
language requested that teachers indicate: (a) what they thought the strengths and 
weaknesses of the required textbook series were with regard to oral production 
activities, (b) whether they used the series’ activities the way they were designed and 
why, (c) what kinds of changes they made to the books’ oral activities and why, (d) 
whether the textbooks lent themselves to these changes, and (e) whether they provided 
additional oral activities for their classes and why (see Beliefs/Materials Questionnaire, 
Appendix A). The textbook series being used was the Spectrum ESL Series Volumes 
1-4 (Dye & Frankfurt, 1993-1994) and Volume 5 (Costinett & Byrd, 1994). 

As with the Beliefs Group, the Materials Group t teacher researchers used 
responses from the first, open-ended questionnaire to construct a second one. Their 
goal was to obtain more information from teachers regarding the strengths and 
weaknesses of specific oral activities appearing in the textbooks (see Beliefs/Materials 
Follow-up Questionnaire, Appendix B). They created a list of nine strengths and eleven 
weaknesses supplied by respondents to the first questionnaire and asked teachers to 
indicate the extent to which they agreed with these judgments about the strengths of the 
activities appearing in their textbooks. Teachers were also asked to indicate the extent 
to which they felt that each activity actually related to oral production. 

An interview was designed to focus on how the teachers carried out specific oral 
activities in the required textbooks in order to gather more detailed information on what 
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was actually happening in the classroom (see Materials Interview, Appendix C). The 
three teacher researchers in the Materials Group chose two representative book 
activities for each level so that they were able to ask teachers about the level they 
taught most frequently. The intention was to show the teacher the book task, ask them 
how they taught the exercise, and ask probing questions if responses lacked detail. 

The teacher-researchers were concerned with what the teachers did first, what 
instruction they gave students, how the students responded, the kinds of materials they 
used for the exercise, the time they spent on the exercise, and how they wrapped it up. 

Assessment Questionnaires and Interview 

The teacher researchers focusing on language assessment (henceforth “the 
Assessment Group”) designed an initial questionnaire which asked the teachers to list 
the features that they considered when assessing students’ oral production and to rank 
these features from most important to least important. Then the questionnaire provided 
a list of possible tasks for assessing students orally -- such as describing an object or 
picture and performing a dialog (based on Brown & Yule, 1983), and requested the 
teachers to indicate for students at the beginning, intermediate, and advanced levels, 
the number of times per quarter they typically did each task. In addition, teachers were 
to indicate the extent to which feedback was given to individuals, pairs/groups, or the 
whole class. Within each of these categories, they were asked if the feedback was 
given in a written form (i.e., through a journal, a note, or an evaluation form) or orally 
(i.e., in conjunction with a form or on tape). Finally, the teachers were asked to indicate 
how often they gave students feedback on their oral production: after each oral task, 
once a week, three times during the term, in the middle and end of the term, at the end 
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of the term, or when needed (see Assessment Questionnaire, Appendix A). 

In analyzing the responses to the Assessment Group's survey, the teacher 
researchers found that teachers in their sample had misunderstood their third question 
about methods of feedback on oral production. Many teachers thought that written 
feedback on students’ oral production actually meant feedback on students’ written 
work. Therefore, the investigators decided to clarify this and ask the question again in a 
follow-up questionnaire (see Assessment Follow-up Questionnaire, Appendix B). They 
also realized that it would be useful to know which methods of feedback were the most 
and least used, so in addition they asked teachers to identify the methods that they 
used the most and least frequently. On the question about the frequency of feedback 
on oral production in the initial questionnaire, many teachers had given more than one 
response when asked to give only one. Therefore, this question was clarified and also 
included in the follow-up questionnaire. 

The three Assessment Group teachers focused the design of an Assessment 
Interview on gathering more details on how teachers assessed students orally (see 
Assessment Interview, Appendix C). The interview included questions on: (a) how they 
decided on the number of times to implement an oral assessment task, (b) how they 
chose which tasks to use with a particular class, (c) how feedback to students regarding 
their performance on an oral assessment task was given, (d) how they decided on how 
often to give the students feedback, and (5) what step-by-step procedure was employed 
when giving students feedback. 



Data Collection and Analysis Procedures 

Since pilot results had revealed that the Colombian teacher researchers had a 
tendency to be overly unstructured in a research design that invariably called for a 
structured approach to research, it was decided to devise strict procedures and scripts 
for all interactions with subjects (e.g., requests for participation, giving of directions, 
reminders to participants, and the like) in order to minimize the researcher effect. 

Teacher Questionnaires and Interviews 

The questionnaires for the Beliefs and Materials Groups were fairly short, so they 
were combined and distributed to 25 teachers. Over two-thirds of these teachers 
responded (68%). Interviews by the Materials Group were conducted with a random 
sub-sample of over half the teachers from the original group (56%) subsequent to their 
responding to the questionnaire and an interview was also conducted with the Adult 
English Program Coordinator. The Oral Assessment questionnaire was given to the 
other 26 teachers and 77% were returned. In addition, 42% of the teachers who 
received this Oral Assessment questionnaire were asked to participate in interviews. 

The two teacher questionnaires (Materials/Beliefs and Oral Assessment) were 
distributed randomly in a mandatory teachers’ meeting where the project and its 
benefits to the teachers were explained. Teachers were also asked to sign a consent 
form at this meeting. The Follow-Up Questionnaires for the Beliefs Group and the 
Materials Group were again combined and distributed to twenty-two of the original 
sample of twenty-five teachers at the Language Center. The Follow-Up Assessment 
Questionnaire was distributed to twenty of the twenty-six teachers who had been given 
the first Assessment Questionnaire. Eleven teachers responded to this follow-up 
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questionnaire. 

Student Questionnaire 

For the student Beliefs Questionnaire, the researchers visited classes to explain 
the project, ask students to participate, and have them sign consent forms. The 
students filled out the questionnaires in class, with a researcher present to answer any 
questions and collect the questionnaires. Again, strict procedures and scripts were 
adhered to in the collection of the student data and in responding to students' questions. 
The procedures indicated the steps to take both before and during the interview, as well 
as providing written instructions for the interviewer to use. 

Classroom Observation 

The Beliefs Group teacher researchers decided to observe six different classes 
for six consecutive hours each. Of the six classes observed, four were at the beginning 
level, one at the intermediate, and one at the advanced, reflecting the relative 
distribution of students across levels. Before the observations, the researchers went to 
the classes involved to explain what would be happening, why the class was being 
videotaped, and to get consent from both the students and their teacher. Procedures 
and scripts were written for this process as well. The reason for observing six 
consecutive hours was that this was the amount of time allotted for completing a unit, 
and researchers felt that within a unit, all types of oral activities were typically 
represented. In order to compare actual oral language activity in the classroom with 
teachers’ and students’ beliefs about the amount of teacher talk and student talk there 
should be, the group videotaped the observed classes and timed the amount of teacher 



and student talk. 



Data Analysis Procedures 4 

Simple frequency counts for the raw data and/or percentages by category were 
used in the reporting of the findings because of the small sample size and unequal data 
entries across categories. 



Results 

1. How do teachers and students believe oral instruction should be handled in 
the classroom? To what extent do the beliefs that teachers have about students’ 
oral production in the classroom and the students’ own beliefs match the reality? 

Amount of Teacher Talk and Student Talk 

The Belief Group's first question concerned the amount of teacher talk that 
students and teachers felt was ideal. While the majority of both teachers and students 
believed teachers were to talk more than the students, students felt this even more 
strongly. The reasons given most frequently by teachers were: “in order to provide 
instruction," "to provide students with an opportunity to practice,” and “to give students 
feedback.” Students also felt that providing instruction and having an opportunity to 
practice were very important. However, another reason given by students was “to 
develop their language skills.” Regarding the amount of student talk the teachers and 
students felt was ideal, most teachers and students felt that students’ oral participation 
should be at a moderate level. 

When looking only at the amount of time that involved the teacher and/or 
students talking (excluding the other activities), 57% comprised teacher talk and 43% 

4 We gratefully acknowledge the efforts of Dr. Marta Eugenia Alvarez Valle, a professor 
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student talk or group work. This was consistent with what students believed should 
happen in the classroom, but less than what teachers believed was appropriate. Some 
teachers believed that students should be talking as much as 80% of the time. 

Characteristics of Good Oral Production 

With regard to what teachers and students believed were the characteristics of 
good oral production, there were both similarities and differences in teachers' and 
students’ responses (see Table 1). Forty-seven percent of the teachers believed that 
good grammar was a major aspect of good oral production, whereas only 18% of the 
students believed this. In contrast, 40% of the students believed fluency to be important 
vs. 29% of the teachers. Seventeen percent of the students believed listening 
comprehension to be important while only 6% of the teachers did. 

In addition, there was also a disparity between the oral production goals as 
articulated by the Director and Academic Committee of the Language Center and the 
teachers’ beliefs as to what constituted good oral production. While the articulated goal 
of the Language Center, as noted above, was to emphasize fluency and meaning, 
teachers tended to value form and accuracy in oral language as can be seen by their 
ranking "grammar” as the most important, and “vocabulary" and “pronunciation” second 
of the items they listed as characteristics of good oral production (see Table 1). 

Beliefs about Range of Oral Activities and Classroom Observations 
In comparing the results of the beliefs questionnaires with the data collected in 
the class observations, it was found that there was a sizeable difference between the 
activities that teachers listed as appropriate in the questionnaires and what was actually 



of statistics from EAFIT University, in performing the statistical analyses. 
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observed in the classroom. Although teachers felt that a great variety of activities were 
appropriate for learning English, few were actually employed in the classroom; 46% of 
the oral activities observed were “question and answer" activities. While two-thirds of 
the teachers had indicated on the questionnaire that whole-class work was either 
“appropriate” or “very appropriate,” many more than that (89%) employed this class 
arrangement in the oral activities that were observed. In addition, whereas ninety-five 
percent of teachers felt that pair work was appropriate and 100% group work, only 8% 
of the oral activities observed were carried out in pairs and 3% in groups. Finally, the 
classroom observations revealed that there was an average of three oral activities per 
100-minute class, lasting for an average of 8.5 minutes. That meant that only one- 
quarter of the class time comprised oral activities. Hence, there was a conflict between 
the communicative approach that the Language Center aimed to employ and the limited 
role of communication as observed by means of the videotaping. 

2. How are teachers using the required materials for oral language production? 

Strengths and Weaknesses of the Required Textbook Series 

The following five strengths of the Spectrum series emerged from the teacher 
questionnaire responses: (1) promotes oral language use, (2) has realistic situations in 
dialogs, (3) facilitates the expression of ideas, opinions, and feelings, (4) promotes 
communication from beginning levels, and (5) promotes oral interaction. However, it 
was found that teachers did not come to consensus on what the textbooks’ strengths 
and weaknesses were. 




18 



19 



How Teachers Used the Oral Activities in the Textbooks 
Teachers indicated that they sometimes altered the way they used lessons from 
the textbook series in order to make oral activities more student-centered and to make 
the book situations more meaningful and realistic. In addition, some teachers indicated 
attempting to make the lessons more communicative or to give the students an 
opportunity to practice the language. A third of the teachers noted that adaptations 
were easy to make because the books’ oral activities could be related to the students’ 
lives. 

3. How do teachers assess students orally? 

Characteristics Considered When Assessing Students Orally 

Regarding the characteristics that teachers considered when assessing students 
orally and the importance given to each characteristic, the top two characteristics 
generated by teachers were found to be pronunciation and grammar. The fact that all of 
these characteristics were based on accuracy runs counter to a communicative 
approach to teaching. One of the elements emphasized in a communicative classroom, 
making oneself comprehensible, was ranked last out of nine characteristics, along with 
discourse (see Table 2). 

Tasks for Assessing Students Orally 

The teachers reported using a wide variety of tasks in oral assessment of 
students. This was true across levels as well. In the Spectrum series (Dye & Frankfurt, 
1993-1994; Costinett & Byrd, 1994), description, role-play, and dialog activities were 
found throughout. Activities involving the "giving of instructions" were more prevalent in 
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the lower-level texts, and activities involving "narration" and "giving opinions" more in 
the upper level texts. It would appear, therefore, that teachers tended to use the types 
of tasks presented in their texts, possibly meaning that the texts, rather than the 
teachers, were dictating the types of tasks used. 

Feedback Method for Oral Production 

With regard to the methods teachers employed for giving students feedback on 
their oral production, the results indicated that when given to individuals, pairs, or small 
groups, there was reported variety in whether feedback was presented orally or involved 
some written format, such as an evaluation sheet. However, when given to the class as 
a whole, the teacher most frequently reported delivering feedback orally. 

Discussion and Conclusions 

Summary 

The Beliefs Group researchers found that while there was general agreement 
between student and teacher beliefs regarding the appropriate amount of student and 
teacher talk in the classroom, teachers believed in more robust student participation. In 
comparing these beliefs to actual classroom observation, there was an observed 
discrepancy in that teacher talk dominated the observed classes. In addition, teacher 
beliefs and the reality did not always match the Language Center’s oral language goal, 
which was to “enable students to communicate orally through a communicative 
approach to teaching: providing student-centered courses, encouraging interaction in 
the classroom through pair work and group work, and presenting a variety of 
opportunities for students to produce spoken language.” 



The findings from the Materials Group indicated that the teachers in the study 
had differing opinions regarding the strengths and weaknesses of their textbook series, 
and employed a variety of methods for making the textbooks work for them. However, 
because the types of changes teachers were making most frequently would have 
needed to be made with any textbook in order to meet the needs of the specific 
population being taught, and because teachers felt that it was easy to make these 
changes, it would seem that the series was serving its purpose. On the basis of these 
data, therefore, the researchers concluded that the series did not need to be replaced. 

Assessment Group researchers found that teachers reported focusing on 
pronunciation and grammatical accuracy when assessing students’ oral abilities, rather 
than on more communicative aspects of oral production (e.g., fluency, making oneself 
comprehensible, and discourse). In addition, there was no prevalent method across 
teachers for giving feedback, with a preference for using the assessment tasks provided 
in the textbook. 

Limitations 

The researchers faced some limitations in drawing these conclusions. In many 
cases the sample size was limited. With the Beliefs Group, the large difference in 
teacher and student sample sizes made it difficult to compare the two groups 
numerically. Another limitation was that the teacher population was not fully sampled 
because of the difficulty of getting questionnaires back from teachers. In addition, even 
when questionnaires were filled out, it appeared that both teachers and students did not 
necessarily understand the wording of items on the pilot questionnaire and sometimes 
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even on the revised versions. In some ways, it was a challenge for the teachers to be 
doing the research in a South American city where local perspectives on what 
constituted research and how to conduct it were sometimes at odds with conventional 
approaches in the Western world, and where there was no means for compensating 
participants since this was an idea foreign to the local culture. 

Still another limitation was that because the questionnaires were anonymous, it 
was not known whether the six teachers who were observed teaching had filled out the 
Beliefs questionnaire. Had they done so, it may have had some impact on their 
observed behavior (such as the amount of their teacher-talk and types of oral activities). 

Suggestions for Future Research and Pedagogical Implications 

It would seem important to investigate why the fit between teachers’ beliefs and 
classroom instructional practice was not closer. It was evident from the information 
gathered by the Beliefs Group that both teachers and students could benefit from a 
better sense of what a "communicative" classroom actually entails. In addition, perhaps 
teachers could benefit from training in how to apply their beliefs to their classroom 
practice. The main pedagogical implications of the study were twofold: (1) there was a 
need to provide guidance both to teachers and students in how to make EFL teaching 
and learning truly communicative in nature, and (2) teachers needed more training in 
how to conduct oral language assessment in the classroom. 
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Conclusions 

One of the strengths of this study was its efforts at convergent validation by 
having three different groups of teacher researchers converging on the same issue, 
namely oral language instruction and assessment from different vantage points. 

Overall, what the research on oral language production at the EAFIT University 
Language Center showed was that although the program claimed to have a 
communicative approach to teaching, the teachers had not been completely successful 
in implementing this approach. 

Some of the data gathered, especially from the Materials Group, showed that 
teachers were aware of the elements of a communicative classroom and were trying to 
implement this approach when teaching. However, they had not applied these concepts 
to all areas of their classes, as was especially evident in the data gathered by the 
Assessment Group. Moreover, the beliefs held by teachers and by students did not 
generally reflect a communicative approach to second language teaching. What this 
meant for the Language Center was that more training of the teachers and more 
education of both teachers and students needed to be done in order to have a truly 
communicative language program. 
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Table 1 



Characteristics of Good Oral Production Listed by Teachers and Students 



Characteristic 5 


Percentage of 
Teachers 


Percentage of 
Students 


Good grammar 


47 


18 


Good vocabulary 


29 


30 


Good fluency 


29 


40 


Good pronunciation 


29 


46 


Effective discourse 


24 


3 


Effective communication 






of message 


18 


8 


Staying in the foreign language 


12 * 


19 


Ability to converse 


12 


5 


Creativity 


6 


3 


Responding appropriately to questions 


6 


3 


Good aural comprehension 


6 


17 



Note : Percentages total more than 100 because subjects could indicate more than one 
characteristic. 



5 The characteristics are actually the investigators' characterization of teacher and student open-ended 
responses. 
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Table 2 

Means and Rankings for Characteristics Teachers Consider 





When Assessing Students Orally 


Overall 

Ranking 


Characteristic Mean Teachers’ Rankings (N=20) 


1 


Pronunciation 


2.70 


2 


Grammar 


2.05 


3 


Vocabulary 


1.53 


4 


Fluency 


1.47 


5 


Learner Strategy Variables 


1.05 


6 


Interaction/Participation 


0.92 . 


7 


Learner Style Variables 


0.89 


8 


Aural Comprehension 


0.74 


9.5 


Making Oneself Comprehensible 0.53 


9.5 


Discourse 


0.53 



Appendix A 



Beliefs/Materials Questionnaire 

The Language Center is conducting research on ways to foster oral language production in 
the classroom. As part of the research project, we would like your input through this 
questionnaire. It is made up of two sections. The first one, in English, is about the oral 
activities in the Spectrum Series. The second one, in Spanish, seeks information on the 
beliefs teachers have about oral instruction. Your responses will help us improve the quality 
of the teaching-learning process at the Language Center. Each section will take you no 
longer than 20 minutes. Put the completed questionnaire in the box labeled “Research 
Questionnaire #1" in the Resource Center. Thank you very much for your cooperation. 

DIRECTIONS: For each question, please explain your answers in detail. If you need further 
information or if you have any questions, please contact Lydia Fass. 



PART I: SPECTRUM SERIES 



^Please answer this part in English. 

1 . Think about the materials you have used from the Spectrum Series. What are their 
strengths and weaknesses in relation to providing oral production activities? 

STRENGTHS of the Spectrum Series: 

WEAKNESSES of the Spectrum Series: 

2. Do you use the oral activities in the books the way they are designed? 

Yes No 

Why or why not? 



What kinds of changes do you make to the books' oral activities? 



Why do you make these changes? 

Do the textbooks lend themselves to these adaptations? Yes No_ 

Why or why not? 

Do you provide additional oral activities for your classes? Yes No 

Why or why not? 



3 . 



PART II - TEACHERS’ BELIEFS ON ORAL INSTRUCTION 



oEsta parte de la encuesta puede responderla en ESPANOL. 

Para responder las preguntas de esta seccion, NO debe tener en cuenta las 
situaciones REALES de la clase, sino lo que usted crea que seria lo IDEAL. Responda 
con honestidad. Sus criterios son de mucha importancia para los resultados de la 
encuesta. 

1. Teniendo en cuenta que cada clase es diferente, en su opinibn, cubl seria el porcentaje 
(%) ideal del tiempo de clase, que deberia hablar el profesor? % 

Por que es apropiado este porcentaje? 



2. Teniendo en cuenta que cada clase es diferente, en su opinibn, cubl seria el porcentaje 
(%) ideal del tiempo de clase, que deberlan hablar los estudiantes ? % 

Por que es apropiado este porcentaje? 



3. Cubles cree usted que son las caracterlsticas de una buena produccibn oral de los 
estudiantes en clase? 

4. En su opinibn, qub tipo de actividades orales son apropiadas para aprendery 
practicar el Ingles en clase? 
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Assessment Questionnaire 



The Language Center is conducting research on ways to foster oral language production in the 
classroom. As part of the research project, we would like your input on how you assess students 
orally. Please take your time to respond to this survey as the information you give us will help us 
improve the assessment process at the Language Center. The survey will take you approximately 20 
minutes. When answering the questions think not only of the courses you are teaching now, but also 
about the courses you have taught in the past. If you have any questions, see Lydia Fass. Put the 
completed questionnaire in the box labeled "Research Questionnaire #2" in the Resource Center. 
Thank you very much for your help. 

1 . Characteristics or features to be considered when assessing students orally 

A. In order to help you think of the features of oral assessment, think about a specific class of 
yours. Which students would you consider to be the best orally? Why? What characteristics 
of their oral production affect that assessment? 

B. In the left hand column list the characteristics or features of their oral production that you 
would use to rank their oral abilities. 

C. In the right hand column rank these characteristics from most important (1) to least important. 



CHARACTERISTICS OR FEATURES TO BE CONSIDERED WHEN 
ASSESSING STUDENTS ORALLY 


RANKING 


















Comments: 



2. Tasks for assessing students orally 

Here is a list of oral assessment tasks for beginner (levels 0-5), intermediate (levels 6-10) and 
advance courses. For each task specify the level in which you have' used it. Also, write the 
NUMBER OF TIMES (not percentage) you have used it per level per quarter. Please write 
answers for both courses you are teaching now, and ones you have taught in the past. 



TASKS Students do 


LEVEL(S) 


Number of times per 
level per quarter 


The student describes an object or picture 


beginner 

intermediate 

advanced 




The student instructs someone to draw something 


beginner 

intermediate 

advanced 




The student instructs someone how to use a piece of 
equipment 


beginner 

intermediate 

advanced 




The student describes/instructs how a number of 
objects are to be arranged 


beginner 

intermediate 

advanced 




The student gives route directions 


beginner 

intermediate 

advanced 
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The student tells a story 



The student gives an eye-witness account 



The student expresses an opinion or idea on a specific 
topic 



beginner 

intermediate 

advanced 



beginner 

intermediate 

advanced 



The student justifies why he/she chooses to do 
something in a certain way. 



The student describes a personal experience 



The student role plays 



The student performs a dialogue 



Other (please specify): 



beginner 

intermediate 

advanced 



beginner 

intermediate 

advanced 



beginner 

intermediate 

advanced 

beginner 

intermediate 

advanced 



beginner 

intermediate 

advanced 

beginner 

intermediate 

advanced 



Comments: 



3. Feedback method for oral production 

The following are ways in which teachers can give students feedback. In the right hand column 
put a check (V) next to each of the methods you use to give students feedback on their oral 
production. 



PEOPLE 


ORAL / WRITTEN 


FORM OF FEEDBACK 


Check (V) here 


RECEIVING 






for the one(s) 


FEEDBACK 






you use 


INDIVIDUAL 


ORAL 


with form 








without form 








taped 








other (specify): 






WRITTEN 










with form 








through journal 








as a note 








other (specify): 
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o 2 

me 



PAIR/GROUP 



ORAL 



WRITTEN 



with form 
without form 
taped 

other (specify):. 



with form 
through journal 
as a note 
other (specify):. 



WHOLE 

CLASS 



ORAL 



WRITTEN 



with form 
without form 
taped 

other (specify):. 



with form 
through journal 
as a note 
other (specify):. 



COMMENTS: 



4. Frequency of feedback on oral production 

Think about how frequently you give students feedback on their oral production in the classes 
you teach. For each level, check the frequency with which you give students feedback on their 
oral production. ‘Beginner’ refers to levels 0-5; 'intermediate' to levels 6-10. 



FREQUENCY 



After each oral task 



LEVEL(S) 

beginner 

intermediate 

advanced 



PUT A CHECK WHERE 
APPROPRIATE 



Once a week 



beginner 

intermediate 

advanced 



In the middle and at the end of every cycle 



beginner 

intermediate 

advanced 



At the end of every cycle 



beginner 

intermediate 

advanced 
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Three times during the cycle 


beginner 

intermediate 

advanced 




When needed 


beginner 

intermediate 

advanced 




Other (s) specify: 


beginner 

intermediate 

advanced 




COMMENTS: 
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Appendix B 

Beliefs/Materials Follow-up Questionnaire 



SURVEY FOR THE EAFIT LANGUAGE CENTER RESEARCH PROJECT 

As you already know there are a number of tools that the research team is using to 
collect data. The first was the questionnaire for teachers. The questionnaires 
provided us with general information. The second phase of data collection involved 
interviewing teachers for more detailed information and observing classes. 

From the information teachers gave us in the interviews, we created a list of 
strengths and weaknesses concerning the Spectrum Series. 

PART I 

We would like you to look at the list and mark whether you completely agree, 
somewhat agree, or do not agree at all that the item is a strength or weakness. For 
each item, please put a check in the appropriate column. 



STRENGTHS 


Completely 

Agree 


Somewhat 

Agree 


Do Not 
Agree at All 


1 . Promotes oral language use 








2. Clearly explains features of 
dialogues 








3. Has realistic situations in 
dialogues 








4. Has language appropriate for 
all levels of formality 








5. Facilitates the expression of 
ideas, opinions and feelings 
through activities 








6. Sequences speaking tasks 
logically (structured to 
unstructured) 








7. Promotes communication from 
beginning levels 








8. Promotes oral interaction 








9. Provides authentic situations 
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WEAKNESSES 


Completely 

Agree 


Somewhat 

Agree 


Do Not 
Agree at All 


1 . Contains mechanical practice 








2. Is boring 








3. Is repetitive 








4. Doesn’t contain authentic 
situations 








5. Is written for English as a 
second language (ESL), not 
English as a foreign language 
(EFL) 








6. Lacks learning strategies focus 








7. Doesn’t contain enough 
explanation concerning 
structures 








8. Contains repetitive structure 
exercises 








9. Has too many structure 
exercises per unit 








10. Has too few activities 








1 1 . Has too many topics 









PART II 



Now please go back to each list (strengths and weaknesses) and put a star to the left of 
the items you think are related to ORAL PRODUCTION. 



Partlll 

The following is a list of oral activities that you told us were appropriate for learning and 
practicing English orally in class. For each item in the list, please mark how appropriate 
you think the activity is for oral practice in the classroom. 





Very 

Appropriate 


Appropriate 


Somewhat 

Appropriate 


Not Very 
Appropriate 


Not 

Appropriate 


Activity 












Roleplay 












Question and answer 












Repetition 












Games 












Pair work 












Group work 












Whole class activities 
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Reporting 












Description 












Telling a story 












Giving instructions 












Discussion 












Debate (formally arguing pros 
and cons of an issue) 












Presentations 












Interviews 












Practicing dialogues 












Oral reading 












Problem solving 












Giving/defending opinions 












Songs 












Substitution activities (drills) 












Summarizing 












Retelling 













Assessment Follow-up Questionnaire 



May 13, 1999 
Dear Teachers, 

We really appreciate your having answered the research questionnaire. We found that 
in order to make the best use of the information you gave us, we need to ask you a few 
more questions. 

PART I 

On question number 3, we would like to clarify what we meant by oral and written 
feedback on students’ oral production. An example of oral feedback would be sitting 
down with your students and discussing their oral performance. An example of written 
feedback would be giving your students a note or an evaluation form that tells them how 
they did during a speaking activity. In the third column, “with form” means using an 
evaluation form. If this is not how you answered question number 3, please go back to 
the original question (attached - page 3) and make any necessary changes to your 
answers. It is okay to erase or cross out the old answers if they are incorrect. 

PART II 

Additionally, we would like to get some more information from you on how you give 
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feedback. On the same chart for question number 3, in the last column where you put 
the checks, put a (+) next to the manner of feedback you use the most and a (-) next to 
the manner of frequency you use the least. 



PART III 

On question number 4, many teachers gave multiple answers but we are looking for 
only one answer per level. We have attached a new copy of this question (page 2). 
Please take a look at the new directions and answer this question again. 

Thank you again. 



4. Frequency of feedback on oral production 

A. Think about how frequently you give systematic feedback to every student in your class; for 
example, going through categories on an evaluation form with them. For each level, check 
how frequently you give students feedback. Check only ONE answer per level. 



FREQUENCY 



LEVEL(S) 



PUT A CHECK WHERE 
APPROPRIATE 



After each oral task 



beginner 

intermediate 

advanced 



Once a week 



Three times during the cycle 



beginner 

intermediate 

advanced 



beginner 

intermediate 

advanced 



In the middle and at the end of every cycle 



At the end of the cycle 



Other (s) specify: 



beginner 

intermediate 

advanced 



beginner 

intermediate 

advanced 



beginner 

intermediate 

advanced 



COMMENTS: 



B. If you give additional feedback when needed, check here: 
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Appendix C 

Materials Group Interview 

Introduction: 

This interview is part of the Research Project on materials being done at the Language 
Center at Eafit. It is the next step after having answered the questionnaire on the 
Spectrum Series. It will take about 30 minutes to complete this interview. Please, 
explain your answers in detail. Thank you very much for your cooperation. 

Interview Question: 

How do you teach this exercise? (refer to a specific exercise from the book) 
SUB-QUESTIONS: 

■ What do you do first? 

■ What instructions do you give your students? 

■ What do you do as a teacher? 

■ What do the students do? 

■ What kind of material do you use for this exercise? 

■ How much time do you spend on this exercise? 

■ How do you wrap-up this exercise? 



Assessment Group Interview 

1 . How do you decide on the number of times you implement an oral assessment 
task with a class? (refer to chart for question 2 of questionnaire) 

2. How do you choose which tasks to use with a particular class? 

3. How do you give your students feedback on their oral production? (refer to 
question 3 of questionnaire) 

4. How do you decide how often to give students feedback on their oral production? 
(refer to question 4 of questionnaire) 

5. Can you describe the procedure you use to give your students feedback? (ask 
for each manner given in answer to question 3) 
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Appendix D: Observation Tool 



ORAL ACTIVITY 


FREQUENCY 


TOTAL 




CLASS 

WORK 


PAIR 

WORK 


SMALL 

GROUP 




Describing (person, object, place, process, event, 
etc.) 










Debating (arguing two sides of an issue) 










Dramatizing (with given or created script - 
forma! presentation.) 










Expressing an opinion or idea (as the focus of 
the activity.) 










Games (purpose of fun or competition.) 










Justifying something (defending a decision or 
action.) 










Oral reading 










Practicing a given dialog 










Presentations & speeches (improvised or 
rehearsed - conveying info.) 










Question & answer 










Repeating (after tape or Teacher.) 










Reporting (giving an account of something seen, read, 
done or heard.) 










Role-playing (with no script) 










Songs 










Story telling 










Student giving instructions (as the focus of the 
activity.) 










Others (specify) 





















OBSERVER’S COMMENTS: 
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